Masked Student Dataset of Expressions

Facial expression recognition (FER) algorithms work well in constrained environments with little or no occlusion of the face. However, real-world face occlusion is prevalent, most notably with the need to use a face mask in the current Covid-19 scenario. While there are works on the problem of occlusion in FER, little has been done before on the particular face mask scenario. Moreover, the few works in this area largely use synthetically created masked FER datasets. Motivated by these challenges posed by the pandemic to FER, we present a novel dataset, the Masked Student Dataset of Expressions or MSD-E, consisting of 1,960 real-world non-masked and masked facial expression images collected from 142 individuals. Along with the issue of obfuscated facial features, we illustrate how other subtler issues in masked FER are represented in our dataset. We then provide baseline results using ResNet-18, finding that its performance dips in the non-masked case when trained for FER in the presence of masks. To tackle this, we test two training paradigms: contrastive learning and knowledge distillation, and find that they increase the model's performance in the masked scenario while maintaining its non-masked performance. We further visualise our results using t-SNE plots and Grad-CAM, demonstrating that these paradigms capitalise on the limited features available in the masked scenario. Finally, we benchmark SOTA methods on MSD-E. The dataset is available at https://github.com/SridharSola/MSD-E. © 2022 ACM.

An automatic improved facial expression recognition for masked faces.

Automatic facial expression recognition (AFER), sometimes referred to as emotional recognition, is important for socializing. Automatic methods in the past two years faced challenges due to Covid-19 and the vital wearing of a mask. Machine learning techniques tremendously increase the amount of data processed and achieved good results in such AFER to detect emotions; however, those techniques are not designed for masked faces and thus achieved poor recognition. This paper introduces a hybrid convolutional neural network aided by a local binary pattern to extract features in an accurate way, especially for masked faces. The basic seven emotions classified into anger, happiness, sadness, surprise, contempt, disgust, and fear are to be recognized. The proposed method is applied on two datasets: the first represents CK and CK +, while the second represents M-LFW-FER. Obtained results show that emotion recognition with a face mask achieved an accuracy of 70.76% on three emotions. Results are compared to existing techniques and show significant improvement.

Comparative Analysis of AI-Based Facial Identification and Expression Recognition Using Upper and Lower Facial Regions

The COVID-19 pandemic has significantly impacted society, having led to a lack of social skills in children who became used to interacting with others while wearing masks. To analyze this issue, we investigated the effects of masks on face identification and facial expression recognition, using deep learning models for these operations. The results showed that when using the upper or lower facial regions for face identification, the upper facial region allowed for an accuracy of 81.36%, and the lower facial region allowed for an accuracy of 55.52%. Regarding facial expression recognition, the upper facial region allowed for an accuracy of 39% compared to 49% for the lower facial region. Furthermore, our analysis was conducted for a number of facial expressions, and specific emotions such as happiness and contempt were difficult to distinguish using only the upper facial region. Because this study used a model trained on data generated from human labeling, it is assumed that the effects on humans would be similar. Therefore, this study is significant because it provides engineering evidence of a decline in facial expression recognition;however, wearing masks does not cause difficulties in identification.

SCN-SAM: A Modified Self-Cure Network for Facial Expression Recognition under Face Masks

Due to COVID-19, wearing masks has become more common. However, it is challenging to recognize expressions in the images of people wearing masks. In general facial recognition problems, blurred images and incorrect annotations of images in large-scale image datasets can make the model's training difficult, which can lead to degraded recognition performance. To address this problem, the Self-Cure Network (SCN) effectively suppresses the over-fitting of the network to images with uncertain labeling in large-scale facial expression datasets. However, it is not clear how well the SCN suppresses the uncertainty of facial expression images with masks. This paper verifies the recognition ability of SCN on images of people wearing masks and proposes a self-adjustment module to further improve SCN (called SCN-SAM). First, we experimentally demonstrate the effectiveness of SCN on the masked facial expression dataset. We then add a self-adjustment module without extensive modifications to SCN and demonstrate that SCN-SAM outperforms state-of-the-art methods in synthetic noise-added FER datasets. © 2023 Global IT Research Institute (GiRI).

Iris

As an experiment designed to question the boundary, relationship, and identity between human bodies and AI robots, Iris is endowed with independent perception and temperament by implementing facial expression recognition, bio-signal measuring, and emotion synthesis. This temperament is stimulated by PAD emotional model and expressed through algorithmic motions generated in real-Time. Emotions of the Iris are affected by the wearer's feelings through sensors that measure the biological signals of the wearer. This ability to perceive and emote reflects certain connections and differences with the wearer as if showing a split personality. Moreover, Iris noticeably affects interpersonal interactions and relationships by showing different responses to people captured by the camera. These effects were heightened during COVID-19 when our faces were covered by masks, which is, based on perception and synthesis, the Iris performs like a highly specialized organ to augment and replace humans' expressions. © 2023 Owner/Author.

A Deep Learning Approach to Facial Expression Recognition in the Presence of Masked Occlusion

Detecting facial expressions is a vital aspect of interpersonal communication. Automatic facial emotion recognition (FER) systems for detecting and analyzing human behavior have been a subject of study for the past decade, and have played key roles in healthcare, crime detection, and other use cases. With the worldwide spread of the COVID-19 pandemic, wearing face masks while interacting in public spaces has become recommended behavior to protect against infection. Therefore, improving existing FER systems to tackle mask occlusion is an extremely important task. In this paper, we analyze how well existing CNN models for FER fare with masked occlusion and present deep CNN architectures to solve this task. We also test some methods to reduce model overfitting, such as data augmentation and dataset balancing. The main metric used to compare the models is accuracy, and the dataset used here is FER2013. Images from FER2013 were covered by masks using a certain transformation, resulting in a new dataset, MFER2013. From our evaluation and experimentation, we found that existing models need to be modified before they can achieve good accuracy on masked datasets. By improving the architecture of the base CNN, we were able to achieve a significantly improved accuracy. © 2022 IEEE.

A Comparative Approach for Facial Expression Recognition in Higher Education Using Hybrid-Deep Learning from Students' Facial Images

Online education has become increasingly common due to the Covid-19 pandemic. A key difference between online and face-to-face education is that instructors often cannot see their students' facial expressions online. This is problematic because facial expressions can help an instructor gauge engagement and understanding. Therefore, it would be useful to find a mechanism whereby students' facial expressions during an online lecture could be monitored. This information can be used as feedback for teachers to change a particular teaching method or maintain another method. This research presents a system that can automatically distinguish students' facial expressions. These comprise eight expressions (anger, attention, disgust, fear, happiness, neutrality, sadness, and surprise). The data for this research was collected from pictures of 70 university students' facial expressions. The data included 6720 images of students' faces distributed equally among the eight expressions mentioned above, that is, 840 images for each category. In this paper, pre-trained deep learning networks (AlexNet, MobileNetV2, GoogleNet, ResNet18, ResNet50, and VGG16) with transfer learning (TL) and K-fold validation (KFCV) were used for recognizing the facial expressions of students. The experiments were conducted using MATLAB 2021a and the best results were recorded by ResNet18 for F1-score and for AUC curve 99%, and 100% respectively. © 2022 Lavoisier. All rights reserved.

A Systematic Literature Survey of Emotion Detection of Students in E-learning

Affective computing is a part of artificial intelligence, which is becoming more important and widely used in education to process and analyze large amounts of data. Consequently, the education system has shifted to an E-learning format because of the COVID-19 epidemic. Then, e-learning is becoming more common in higher education, primarily through Massive Open Online Courses (MOOCs). This study reviewed many prior studies on bolstering educational institutions using AI methods, including deep learning, machine learning, and affective computing. According to the findings, these methods had a very high percentage of success. These studies also helped academic institutions, as well as teachers, understand the emotional state of students in an e-learning environment. © 2022 IEEE.

A Multi-Modal Deep Learning Approach for Emotion Recognition

In recent years, research on facial expression recognition (FER) under mask is trending. Wearing a mask for protection from Covid 19 has become a compulsion and it hides the facial expressions that is why FER under the mask is a difficult task. The prevailing unimodal techniques for facial recognition are not up to the mark in terms of good results for the masked face, however, a multimodal technique can be employed to generate better results. We proposed a multimodal methodology based on deep learning for facial recognition under a masked face using facial and vocal expressions. The multimodal has been trained on a facial and vocal dataset. We have used two standard datasets, M-LFW for the masked dataset and CREMA-D and TESS dataset for vocal expressions. The vocal expressions are in the form of audio while the faces data is in image form that is why the data is heterogenous. In order to make the data homogeneous, the voice data is converted into images by taking spectrogram. A spectrogram embeds important features of the voice and it converts the audio format into the images. Later, the dataset is passed to the multimodal for training. neural network and the experimental results demonstrate that the proposed multimodal algorithm outsets unimodal methods and other state-of-the-art deep neural network models. © Tech Science Press.

Face-mask-aware Facial Expression Recognition based on Face Parsing and Vision Transformer.

As wearing face masks is becoming an embedded practice due to the COVID-19 pandemic, facial expression recognition (FER) that takes face masks into account is now a problem that needs to be solved. In this paper, we propose a face parsing and vision Transformer-based method to improve the accuracy of face-mask-aware FER. First, in order to improve the precision of distinguishing the unobstructed facial region as well as those parts of the face covered by a mask, we re-train a face-mask-aware face parsing model, based on the existing face parsing dataset automatically relabeled with a face mask and pixel label. Second, we propose a vision Transformer with a cross attention mechanism-based FER classifier, capable of taking both occluded and non-occluded facial regions into account and reweigh these two parts automatically to get the best facial expression recognition performance. The proposed method outperforms existing state-of-the-art face-mask-aware FER methods, as well as other occlusion-aware FER methods, on two datasets that contain three kinds of emotions (M-LFW-FER and M-KDDI-FER datasets) and two datasets that contain seven kinds of emotions (M-FER-2013 and M-CK+ datasets).

Facial expression recognition of masked faces using deep learning

Facial expression recognition (FER) represents one of the most prevalent forms of interpersonal communication, which contains rich emotional information. But it became even more challenging during the times of COVID, where face masks became a mandatory protection measure, leading to the challenge of occluded lower-face during facial expression recognition. In this study, deep convolutional neural network (DCNN) represents the core of both our full-face FER system and our masked face FER model. The focus was on incorporating knowledge distillation in transfer learning between a teacher model, which is the full-face FER DCNN, and the student model, which is the masked face FER DCNN via the combination of both the loss from the teacher soft-labels vs the student soft labels and the loss from the dataset hard-labels vs the student hard-labels. The teacher-student architecture used FER2013 and a masked customized version of FER2013 as datasets to generate an accuracy of 69% and 61% respectively. Therefore, the study proves that the process of knowledge distillation may be used as a way for transfer learning and enhancing accuracy as a regular DCNN model (student only) would result in 46% accuracy compared to our approach (61% accuracy).

A Multi-Modal Deep Learning Approach for Emotion Recognition

In recent years, research on facial expression recognition (FER) under mask is trending. Wearing a mask for protection from Covid 19 has become a compulsion and it hides the facial expressions that is why FER under the mask is a difficult task. The prevailing unimodal techniques for facial recognition are not up to the mark in terms of good results for the masked face, however, a multimodal technique can be employed to generate better results. We proposed a multimodal methodology based on deep learning for facial recognition under a masked face using facial and vocal expressions. The multimodal has been trained on a facial and vocal dataset. We have used two standard datasets, M-LFW for the masked dataset and CREMA-D and TESS dataset for vocal expressions. The vocal expressions are in the form of audio while the faces data is in image form that is why the data is heterogenous. In order to make the data homogeneous, the voice data is converted into images by taking spectrogram. A spectrogram embeds important features of the voice and it converts the audio format into the images. Later, the dataset is passed to the multimodal for training. neural network and the experimental results demonstrate that the proposed multimodal algorithm outsets unimodal methods and other state-of-the-art deep neural network models. © Tech Science Press.

Facial expression recognition of masked faces using deep learning

Facial expression recognition (FER) represents one of the most prevalent forms of interpersonal communication, which contains rich emotional information. But it became even more challenging during the times of COVID, where face masks became a mandatory protection measure, leading to the challenge of occluded lower-face during facial expression recognition. In this study, deep convolutional neural network (DCNN) represents the core of both our full-face FER system and our masked face FER model. The focus was on incorporating knowledge distillation in transfer learning between a teacher model, which is the full-face FER DCNN, and the student model, which is the masked face FER DCNN via the combination of both the loss from the teacher soft-labels vs the student soft labels and the loss from the dataset hard-labels vs the student hard-labels. The teacher-student architecture used FER2013 and a masked customized version of FER2013 as datasets to generate an accuracy of 69% and 61% respectively. Therefore, the study proves that the process of knowledge distillation may be used as a way for transfer learning and enhancing accuracy as a regular DCNN model (student only) would result in 46% accuracy compared to our approach (61% accuracy). © 2023, Institute of Advanced Engineering and Science. All rights reserved.

Contrastive Representation Learning for Expression Recognition from Masked Face Images

With the worldwide spread of COVID-19, people are trying different ways to prevent the spread of the virus. One of the most useful and popular ways is wearing a face mask. Most people wear a face mask when they go out, which makes facial expression recognition become harder. Thus, how to improve the performance of the facial expression recognition model on masked faces is becoming an important issue. However, there is no public dataset that includes facial expressions with masks. Thus, we built two datasets which are a real-world masked facial expression database (VIP-DB) and a man-made masked facial expression database (M-RAF-DB). To reduce the influence of masks, we utilize contrastive representation learning and propose a two-branches network. We study the influence of contrastive learning on our two datasets. Results show that using contrastive representation learning improves the performance of expression recognition from masked face images. © 2022 ACM.

Emotional-Health-Oriented Urban Design: A Novel Collaborative Deep Learning Framework for Real-Time Landscape Assessment by Integrating Facial Expression Recognition and Pixel-Level Semantic Segmentation.

Emotional responses are significant for understanding public perceptions of urban green space (UGS) and can be used to inform proposals for optimal urban design strategies to enhance public emotional health in the times of COVID-19. However, most empirical studies fail to consider emotion-oriented landscape assessments under dynamic perspectives despite the fact that individually observed sceneries alter with angle. To close this gap, a real-time sentimental-based landscape assessment framework is developed, integrating facial expression recognition with semantic segmentation of changing landscapes. Furthermore, a case study using panoramic videos converted from Google Street View images to simulate changing scenes was used to test the viability of this framework, resulting in five million big data points. The result of this study shows that through the collaboration of deep learning algorithms, finer visual variables were classified, subtle emotional responses were tracked, and better regression results for valence and arousal were obtained. Among all the predictors, the proportion of grass was the most significant predictor for emotional perception. The proposed framework is adaptable and human-centric, and it enables the instantaneous emotional perception of the built environment by the general public as a feedback survey tool to aid urban planners in creating UGS that promote emotional well-being.

How facial masks alter the interaction of gaze direction, head orientation, and emotion recognition.

The COVID-19 pandemic has altered the way we interact with each other: mandatory mask-wearing obscures facial information that is crucial for emotion recognition. Whereas the influence of wearing a mask on emotion recognition has been repeatedly investigated, little is known about the impact on interaction effects among emotional signals and other social signals. Therefore, the current study sought to explore how gaze direction, head orientation, and emotional expression interact with respect to emotion perception, and how these interactions are altered by wearing a face mask. In two online experiments, we presented face stimuli from the Radboud Faces Database displaying different facial expressions (anger, fear, happiness, neutral, and sadness), gaze directions (-13°, 0°, and 13°), and head orientations (-45°, 0°, and 45°) - either without (Experiment 1) or with mask (Experiment 2). Participants categorized the displayed emotional expressions. Not surprisingly, masks impaired emotion recognition. Surprisingly, without the mask, emotion recognition was unaffected by averted head orientations and only slightly affected by gaze direction. The mask strongly interfered with this ability. The mask increased the influence of head orientation and gaze direction, in particular for the emotions that were poorly recognized with mask. The results suggest that in case of uncertainty due to ambiguity or absence of signals, we seem to unconsciously factor in extraneous information.

Real-time Facial Expression Recognition via Dense & Squeeze-and-Excitation Blocks

Due to the coronavirus disease 2019 (COVID-19) pandemic, traditional face-to-face courses have been transformed into online and e-learning courses. Although online courses provide flexible teaching and learning in terms of time and place, teachers cannot be fully aware of their students’ individual learning situation and emotional state. The cognition of learning emotion with facial expression recognition has been a vital issue in recent years. To achieve affective computing, the paper presented a fast recognition model for learning emotions through Dense Squeeze-and-Excitation Networks (DSENet), which rapidly recognizes students’ learning emotions, while the proposed real-time online feedback system notifies teacher instantaneously. Firstly, DSENet is trained and validated by an open dataset called Facial Expression Recognition 2013. Then, we collect students’ learning emotions from e-learning classes and apply transfer learning and data augmentation techniques to improve the testing accuracy. The proposed DSENet model and real-time online feedback system aim to realize effective e-learning for any teaching and learning environments, especially in the COVID-19 environment of late © This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Assessing Deep Learning Approaches in Detecting Masked Facial Expressions

Automated facial expression recognition (FER) is an active research area due to its practical importance in a wide range of applications. In recent years, deep learning-based approaches have delivered promising performances in FER, leveraging the latest advances in computer vision. However, mask-wearing after the onset of the COVID-19 pandemic has posed challenges for the existing models when the salient features from the masked region are unavailable. This study investigates what effects facial masks will bring to expression detection using state-of-the-art deep learners. Specifically, we evaluate three deep neural networks in recognizing six emotional categories on masked facial images and compare the results to unmasked counterparts reported by prior studies. We based our work on the FER2013 dataset and augmented regular face images with artificial masks utilizing the Dlib and OpenCV libraries. Our experimental results indicate that deep learning models can be effective in recognizing some masked expressions (e.g., 'Happy', 'Surprise', and 'Neural') but fall short on the others (e.g., 'Angry', 'Fear', 'Sad'). Furthermore, with the presence of facial masks, angry faces are most likely to be misclassified as neural, and fear is the most challenging emotion to detect. © 2022 IEEE.

Evoker: Narrative-based Facial Expression Game for Emotional Development of Adolescents

Evoker is a narrative-based facial expression game. Due to the COVID-19 pandemic, adolescents should be wearing masks in their daily lives. However, wearing masks disturbs emotional interaction through facial expressions, which is a critical component in emotional development. Therefore, a negative impact on adolescent emotional development is predicted. To solve this problem, we design a narrative-based game Evoker that uses real-time facial expression recognition. In this game, players are asked to identify an emotion from narrative contexts in missions, and make a facial expression appropriate for the context to clear the challenges. Our game provides an opportunity to practice reading emotional contexts and expressing appropriate emotions, which has a high potential for promoting emotional development of adolescents. © 2022 Owner/Author.

Collaborative Attention Transformer on facial expression recognition under partial occlusion

Recent research on facial expression recognition (FER) in the wild shows challenges still remain. Different from laboratory-controlled expression in the past, images in the wild contain more uncertainties, such as different forms of face information occlusion, ambiguous facial images, noisy labels, and so on. Among them, real-world facial occlusion is the most general and crucial challenge for FER. In addition, because of the COVID-19 disease epidemic, people have to wear masks in public, which brings new challenges to FER tasks. Due to the recent success of the Transformer on numerous computer vision tasks, we propose a Collaborative Attention Transformer (CAT) network that first uses Cross-Shaped Window Transformer as the backbone for the FER task. Meanwhile, two attention modules are collaborated. Channel-Spatial Attention Module is designed to increase the attention of the network to global features. Moreover, Window Attention Gate is used to enhance the ability of the model to focus on local details. The proposed method is evaluated on two public in-The-wild facial expression datasets, RAF-DB and FERPlus, and the results demonstrate that our CAT performs superior to the state-of-The-Art methods. © 2022 SPIE and IST.